{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <p class=\"display-1 h1\">长短期记忆网络（LSTM）</p>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 为什么需要LSTM？\n",
    "\n",
    "### RNN的局限性回顾\n",
    "\n",
    "在标准RNN中，我们遇到了**梯度消失问题**：\n",
    "- 梯度在反向传播时指数衰减\n",
    "- 难以学习长期依赖（>10个时间步）\n",
    "- 早期时间步的参数几乎不更新\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的提出\n",
    "\n",
    "**长短期记忆网络（LSTM）**由Hochreiter和Schmidhuber在1997年提出，是解决梯度消失问题的最早方法之一。\n",
    "\n",
    "**核心思想**：\n",
    "- 引入**记忆元（Memory Cell）**来长期保存信息\n",
    "- 使用**门控机制**精确控制信息的流动\n",
    "- 通过**遗忘门**和**输入门**管理记忆的更新"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 门控记忆元\n",
    "\n",
    "### 设计灵感\n",
    "\n",
    "LSTM的设计灵感来自于**计算机的逻辑门**：\n",
    "- 就像电路中的开关，可以控制电流的通过\n",
    "- LSTM的门控制信息的流动和存储\n",
    "\n",
    "### 记忆元的概念\n",
    "\n",
    "**记忆元（Memory Cell）**是LSTM的核心组件：\n",
    "- 形状与隐状态相同：`(batch_size, num_hiddens)`\n",
    "- 专门用于**长期保存信息**\n",
    "- 与隐状态分离，提供更灵活的信息管理\n",
    "\n",
    "**类比**：\n",
    "- 隐状态：短期记忆（当前上下文）\n",
    "- 记忆元：长期记忆（重要信息）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的三个门\n",
    "\n",
    "LSTM使用三个门来控制记忆元：\n",
    "\n",
    "1. **输入门（Input Gate）**：决定**何时将新信息写入**记忆元\n",
    "2. **遗忘门（Forget Gate）**：决定**何时遗忘旧信息**\n",
    "3. **输出门（Output Gate）**：决定**何时将记忆元的信息输出**到隐状态\n",
    "\n",
    "**门的作用**：\n",
    "- 每个门都是一个0到1之间的值\n",
    "- 0表示\"关闭\"（不通过）\n",
    "- 1表示\"打开\"（完全通过）\n",
    "- 中间值表示\"部分通过\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 输入门、遗忘门和输出门\n",
    "\n",
    "三个门的计算方式相同，都使用**sigmoid激活函数**：\n",
    "\n",
    "**输入**：\n",
    "- 当前时间步的输入：$\\mathbf{X}_t$\n",
    "- 前一个时间步的隐状态：$\\mathbf{H}_{t-1}$\n",
    "\n",
    "**输出**：\n",
    "- 三个门的值都在$(0, 1)$范围内\n",
    "- 每个门都是形状为$(n, h)$的张量（$n$是批量大小，$h$是隐藏单元数）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src='lstm_1.png' width='800'>\n",
    "\n",
    "**图1：LSTM的三个门**\n",
    "\n",
    "三个门都接收相同的输入（$\\mathbf{X}_t$和$\\mathbf{H}_{t-1}$），但使用不同的权重参数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 三个门的数学公式\n",
    "\n",
    "假设有$h$个隐藏单元，批量大小为$n$，输入数为$d$。\n",
    "\n",
    "**输入门**：\n",
    "$$\\mathbf{I}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{xi} + \\mathbf{H}_{t-1} \\mathbf{W}_{hi} + \\mathbf{b}_i)$$\n",
    "\n",
    "**遗忘门**：\n",
    "$$\\mathbf{F}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{xf} + \\mathbf{H}_{t-1} \\mathbf{W}_{hf} + \\mathbf{b}_f)$$\n",
    "\n",
    "**输出门**：\n",
    "$$\\mathbf{O}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{xo} + \\mathbf{H}_{t-1} \\mathbf{W}_{ho} + \\mathbf{b}_o)$$\n",
    "\n",
    "其中：\n",
    "- $\\sigma$：sigmoid函数，输出范围$(0, 1)$\n",
    "- $\\mathbf{W}_{xi}, \\mathbf{W}_{xf}, \\mathbf{W}_{xo}$：输入到门的权重 $(d \\times h)$\n",
    "- $\\mathbf{W}_{hi}, \\mathbf{W}_{hf}, \\mathbf{W}_{ho}$：隐状态到门的权重 $(h \\times h)$\n",
    "- $\\mathbf{b}_i, \\mathbf{b}_f, \\mathbf{b}_o$：偏置参数 $(h,)$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 为什么使用sigmoid函数？\n",
    "\n",
    "**sigmoid函数的特点**：\n",
    "- 输出范围：$(0, 1)$\n",
    "- 可以看作\"概率\"或\"比例\"\n",
    "- 平滑可导，便于梯度反向传播\n",
    "\n",
    "**在LSTM中的作用**：\n",
    "- 门的值表示\"通过的比例\"\n",
    "- 例如：遗忘门=0.8 表示保留80%的旧信息\n",
    "- 例如：输入门=0.3 表示只采用30%的新信息\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 候选记忆元\n",
    "\n",
    "### 什么是候选记忆元？\n",
    "\n",
    "**候选记忆元（Candidate Memory Cell）** $\\tilde{\\mathbf{C}}_t$：\n",
    "- 是**准备写入记忆元的新信息**\n",
    "- 类似于RNN中的新隐藏状态\n",
    "- 但还没有经过门的筛选\n",
    "\n",
    "**关键区别**：\n",
    "- 候选记忆元：使用$\\tanh$激活函数，值范围$(-1, 1)$\n",
    "- 门：使用$\\sigma$（sigmoid）激活函数，值范围$(0, 1)$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 候选记忆元的计算公式\n",
    "\n",
    "- 候选记忆元基于当前输入和前一个隐状态计算，准备写入记忆元。\n",
    "\n",
    "$$\\tilde{\\mathbf{C}}_t = \\tanh(\\mathbf{X}_t \\mathbf{W}_{xc} + \\mathbf{H}_{t-1} \\mathbf{W}_{hc} + \\mathbf{b}_c)$$\n",
    "\n",
    "**参数说明**：\n",
    "- $\\mathbf{W}_{xc}$：输入到候选记忆元的权重 $(d \\times h)$\n",
    "- $\\mathbf{W}_{hc}$：隐状态到候选记忆元的权重 $(h \\times h)$\n",
    "- $\\mathbf{b}_c$：偏置参数 $(h,)$\n",
    "\n",
    "**为什么用tanh？**\n",
    "- tanh输出范围$(-1, 1)$，有界且中心对称\n",
    "- 有助于梯度流动\n",
    "- 与sigmoid配合使用，形成互补\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 记忆元的更新\n",
    "\n",
    "### 记忆元更新的核心公式\n",
    "\n",
    "记忆元的更新是LSTM的关键：\n",
    "\n",
    "$$\\mathbf{C}_t = \\mathbf{F}_t \\odot \\mathbf{C}_{t-1} + \\mathbf{I}_t \\odot \\tilde{\\mathbf{C}}_t$$\n",
    "\n",
    "**公式解读**：\n",
    "- $\\mathbf{F}_t \\odot \\mathbf{C}_{t-1}$：保留的旧信息（遗忘门控制）\n",
    "- $\\mathbf{I}_t \\odot \\tilde{\\mathbf{C}}_t$：添加的新信息（输入门控制）\n",
    "- $\\odot$：按元素乘法（逐元素相乘）\n",
    "\n",
    "**直观理解**：\n",
    "- 遗忘门决定\"忘记多少旧信息\"\n",
    "- 输入门决定\"记住多少新信息\"\n",
    "- 两者相加得到更新后的记忆元\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src='lstm_2.png' width='800'>\n",
    "\n",
    "**图3：记忆元的更新过程**\n",
    "\n",
    "展示了遗忘门和输入门如何共同控制记忆元的更新。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 极端情况的例子\n",
    "\n",
    "**情况1：完全保留旧信息**\n",
    "- 如果遗忘门$\\mathbf{F}_t = 1$（全1向量）\n",
    "- 输入门$\\mathbf{I}_t = 0$（全0向量）\n",
    "- 则：$\\mathbf{C}_t = \\mathbf{C}_{t-1}$（完全保留旧记忆）\n",
    "\n",
    "**情况2：完全替换为新信息**\n",
    "- 如果遗忘门$\\mathbf{F}_t = 0$\n",
    "- 输入门$\\mathbf{I}_t = 1$\n",
    "- 则：$\\mathbf{C}_t = \\tilde{\\mathbf{C}}_t$（完全替换）\n",
    "\n",
    "**情况3：平衡更新**\n",
    "- 遗忘门和输入门都在0到1之间\n",
    "- 实现新旧信息的平衡混合\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 为什么能解决梯度消失？\n",
    "\n",
    "**关键机制**：\n",
    "- 遗忘门可以设置为接近1，使得记忆元几乎不变\n",
    "- 梯度可以直接通过记忆元传播，不经过tanh的压缩\n",
    "- 这避免了梯度在时间步之间的指数衰减\n",
    "\n",
    "**对比RNN**：\n",
    "- RNN：$\\mathbf{H}_t = \\tanh(\\cdots)$，梯度被tanh压缩\n",
    "- LSTM：$\\mathbf{C}_t = \\mathbf{F}_t \\odot \\mathbf{C}_{t-1} + \\cdots$，梯度可以直接流过\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 隐状态的计算\n",
    "\n",
    "### 隐状态与记忆元的关系\n",
    "\n",
    "**重要区别**：\n",
    "- **记忆元** $\\mathbf{C}_t$：内部状态，不直接输出\n",
    "- **隐状态** $\\mathbf{H}_t$：对外输出，用于预测\n",
    "\n",
    "**计算方式**：\n",
    "$$\\mathbf{H}_t = \\mathbf{O}_t \\odot \\tanh(\\mathbf{C}_t)$$\n",
    "\n",
    "**步骤**：\n",
    "1. 对记忆元应用tanh：$\\tanh(\\mathbf{C}_t)$，将值限制在$(-1, 1)$\n",
    "2. 用输出门控制：$\\mathbf{O}_t \\odot \\tanh(\\mathbf{C}_t)$，决定输出多少信息\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 为什么需要tanh和输出门？\n",
    "\n",
    "**为什么对记忆元用tanh？**\n",
    "- 记忆元的值可能很大（没有上界）\n",
    "- tanh将其压缩到$(-1, 1)$，便于后续处理\n",
    "- 确保隐状态的值在合理范围内\n",
    "\n",
    "**为什么需要输出门？**\n",
    "- 不是所有记忆都需要立即输出\n",
    "- 输出门控制\"何时输出\"和\"输出多少\"\n",
    "- 提供更精细的信息控制\n",
    "\n",
    "**类比**：\n",
    "- 记忆元：大脑中的长期记忆\n",
    "- 隐状态：当前思考的内容\n",
    "- 输出门：决定哪些记忆进入当前思考\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src='lstm_3.png' width='800'>\n",
    "\n",
    "**图4：隐状态的计算**\n",
    "\n",
    "隐状态是记忆元经过tanh和输出门控制后的结果。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的完整计算流程\n",
    "\n",
    "**时间步$t$的完整步骤**：\n",
    "\n",
    "1. **计算三个门**：\n",
    "   - 输入门：$\\mathbf{I}_t = \\sigma(\\cdots)$\n",
    "   - 遗忘门：$\\mathbf{F}_t = \\sigma(\\cdots)$\n",
    "   - 输出门：$\\mathbf{O}_t = \\sigma(\\cdots)$\n",
    "\n",
    "2. **计算候选记忆元**：\n",
    "   - $\\tilde{\\mathbf{C}}_t = \\tanh(\\cdots)$\n",
    "\n",
    "3. **更新记忆元**：\n",
    "   - $\\mathbf{C}_t = \\mathbf{F}_t \\odot \\mathbf{C}_{t-1} + \\mathbf{I}_t \\odot \\tilde{\\mathbf{C}}_t$\n",
    "\n",
    "4. **计算隐状态**：\n",
    "   - $\\mathbf{H}_t = \\mathbf{O}_t \\odot \\tanh(\\mathbf{C}_t)$\n",
    "\n",
    "5. **计算输出**：\n",
    "   - $\\mathbf{O}_t = \\mathbf{H}_t \\mathbf{W}_{hq} + \\mathbf{b}_q$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 从零开始实现\n",
    "\n",
    "### 数据准备\n",
    "\n",
    "我们将使用《时光机器》数据集来训练LSTM语言模型。\n",
    "与RNN的实现类似，我们需要：\n",
    "- 加载数据\n",
    "- 构建词汇表\n",
    "- 创建数据迭代器\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:24:18.324326Z",
     "iopub.status.busy": "2023-08-18T07:24:18.323673Z",
     "iopub.status.idle": "2023-08-18T07:24:21.607373Z",
     "shell.execute_reply": "2023-08-18T07:24:21.606483Z"
    },
    "origin_pos": 2,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### get_lstm_params函数\n",
    "\n",
    "该函数初始化LSTM的所有参数：\n",
    "\n",
    "**设计模式**：\n",
    "- 使用内部函数`three()`生成三个参数（权重1、权重2、偏置）\n",
    "- 为每个门和候选记忆元调用`three()`\n",
    "- 最后添加输出层参数\n",
    "\n",
    "**参数列表**：\n",
    "- 输入门：W_xi, W_hi, b_i\n",
    "- 遗忘门：W_xf, W_hf, b_f\n",
    "- 输出门：W_xo, W_ho, b_o\n",
    "- 候选记忆元：W_xc, W_hc, b_c\n",
    "- 输出层：W_hq, b_q\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 5,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的参数数量对比\n",
    "\n",
    "**LSTM参数数量**：\n",
    "- 4组权重（3个门 + 候选记忆元）：每组$(d \\times h + h \\times h + h)$参数\n",
    "- 总共：$4 \\times (d \\times h + h \\times h + h) + (h \\times vocab\\_size + vocab\\_size)$\n",
    "\n",
    "**相比RNN**：\n",
    "- RNN：$2 \\times (d \\times h + h \\times h + h) + (h \\times vocab\\_size + vocab\\_size)$\n",
    "- LSTM参数数量约为RNN的**2倍**\n",
    "\n",
    "**初始化策略**：\n",
    "- 权重：从$\\mathcal{N}(0, 0.01^2)$采样\n",
    "- 偏置：初始化为0\n",
    "- 小权重有助于初始训练稳定\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### init_lstm_state函数\n",
    "\n",
    "该函数返回LSTM的初始状态：\n",
    "\n",
    "**返回值**：\n",
    "- 元组：`(H, C)`\n",
    "- H：隐状态，形状`(batch_size, num_hiddens)`\n",
    "- C：记忆元，形状`(batch_size, num_hiddens)`\n",
    "\n",
    "**注意**：\n",
    "- 两个状态都初始化为零\n",
    "- 使用元组便于后续解包\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:24:21.612618Z",
     "iopub.status.busy": "2023-08-18T07:24:21.611785Z",
     "iopub.status.idle": "2023-08-18T07:24:21.619127Z",
     "shell.execute_reply": "2023-08-18T07:24:21.618267Z"
    },
    "origin_pos": 7,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def get_lstm_params(vocab_size, num_hiddens, device):\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device)*0.01\n",
    "\n",
    "    def three():\n",
    "        return (normal((num_inputs, num_hiddens)),\n",
    "                normal((num_hiddens, num_hiddens)),\n",
    "                torch.zeros(num_hiddens, device=device))\n",
    "\n",
    "    W_xi, W_hi, b_i = three()  # 输入门参数\n",
    "    W_xf, W_hf, b_f = three()  # 遗忘门参数\n",
    "    W_xo, W_ho, b_o = three()  # 输出门参数\n",
    "    W_xc, W_hc, b_c = three()  # 候选记忆元参数\n",
    "    # 输出层参数\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "    # 附加梯度\n",
    "    params = [W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc,\n",
    "              b_c, W_hq, b_q]\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 代码实现细节\n",
    "\n",
    "**按元素乘法**：\n",
    "- `F * C`：遗忘门控制旧记忆\n",
    "- `I * C_tilda`：输入门控制新信息\n",
    "- `O * torch.tanh(C)`：输出门控制隐状态\n",
    "\n",
    "**输出拼接**：\n",
    "- 所有时间步的输出拼接：`torch.cat(outputs, dim=0)`\n",
    "- 形状：`(num_steps * batch_size, vocab_size)`\n",
    "\n",
    "**状态返回**：\n",
    "- 返回最后一个时间步的状态：`(H, C)`\n",
    "- 用于下一个批次的初始化\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 10,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM前向传播函数详解\n",
    "\n",
    "`lstm`函数实现了LSTM的完整前向传播过程。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 训练结果观察\n",
    "\n",
    "**性能指标**：\n",
    "- 困惑度：通常比RNN更低（更好的预测能力）\n",
    "- 训练速度：比RNN慢（更多参数和计算）\n",
    "\n",
    "**文本生成质量**：\n",
    "- 生成的文本通常更连贯\n",
    "- 能更好地保持长期依赖关系\n",
    "- 语法和语义更合理\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:24:21.623788Z",
     "iopub.status.busy": "2023-08-18T07:24:21.623183Z",
     "iopub.status.idle": "2023-08-18T07:24:21.628423Z",
     "shell.execute_reply": "2023-08-18T07:24:21.627603Z"
    },
    "origin_pos": 12,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def init_lstm_state(batch_size, num_hiddens, device):\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device),\n",
    "            torch.zeros((batch_size, num_hiddens), device=device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 15,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### lstm函数的实现步骤\n",
    "\n",
    "该函数实现了LSTM的完整前向传播，对每个时间步执行：\n",
    "\n",
    "**步骤1：计算三个门**\n",
    "```python\n",
    "I = torch.sigmoid(...)  # 输入门\n",
    "F = torch.sigmoid(...)  # 遗忘门\n",
    "O = torch.sigmoid(...)  # 输出门\n",
    "```\n",
    "\n",
    "**步骤2：计算候选记忆元**\n",
    "```python\n",
    "C_tilda = torch.tanh(...)  # 候选记忆元\n",
    "```\n",
    "\n",
    "**步骤3：更新记忆元**\n",
    "```python\n",
    "C = F * C + I * C_tilda  # 遗忘旧信息，添加新信息\n",
    "```\n",
    "\n",
    "**步骤4：计算隐状态**\n",
    "```python\n",
    "H = O * torch.tanh(C)  # 输出门控制隐状态\n",
    "```\n",
    "\n",
    "**步骤5：计算输出**\n",
    "```python\n",
    "Y = H @ W_hq + b_q  # 只有隐状态用于输出\n",
    "```\n",
    "\n",
    "**重要**：记忆元$\\mathbf{C}_t$是内部状态，不直接参与输出计算。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:24:21.632468Z",
     "iopub.status.busy": "2023-08-18T07:24:21.631955Z",
     "iopub.status.idle": "2023-08-18T07:24:21.638292Z",
     "shell.execute_reply": "2023-08-18T07:24:21.637498Z"
    },
    "origin_pos": 17,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def lstm(inputs, state, params):\n",
    "    [W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c,\n",
    "     W_hq, b_q] = params\n",
    "    (H, C) = state\n",
    "    outputs = []\n",
    "    for X in inputs:\n",
    "        I = torch.sigmoid((X @ W_xi) + (H @ W_hi) + b_i)\n",
    "        F = torch.sigmoid((X @ W_xf) + (H @ W_hf) + b_f)\n",
    "        O = torch.sigmoid((X @ W_xo) + (H @ W_ho) + b_o)\n",
    "        C_tilda = torch.tanh((X @ W_xc) + (H @ W_hc) + b_c)\n",
    "        C = F * C + I * C_tilda\n",
    "        H = O * torch.tanh(C)\n",
    "        Y = (H @ W_hq) + b_q\n",
    "        outputs.append(Y)\n",
    "    return torch.cat(outputs, dim=0), (H, C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 20,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 使用RNNModelScratch类\n",
    "\n",
    "我们复用`RNNModelScratch`类来训练LSTM：\n",
    "\n",
    "**类的优势**：\n",
    "- 统一的接口：可以用于RNN、GRU、LSTM\n",
    "- 函数式设计：通过传入不同的函数实现不同模型\n",
    "- 代码复用：训练和预测代码无需修改\n",
    "\n",
    "**关键参数**：\n",
    "- `get_lstm_params`：LSTM的参数初始化函数\n",
    "- `init_lstm_state`：LSTM的状态初始化函数\n",
    "- `lstm`：LSTM的前向传播函数\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:24:21.642387Z",
     "iopub.status.busy": "2023-08-18T07:24:21.641860Z",
     "iopub.status.idle": "2023-08-18T07:28:54.778673Z",
     "shell.execute_reply": "2023-08-18T07:28:54.777826Z"
    },
    "origin_pos": 21,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.3, 17736.0 tokens/sec on cuda:0\n",
      "time traveller for so it will leong go it we melenot ir cove i s\n",
      "traveller care be can so i ngrecpely as along the time dime\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"180.65625pt\" viewBox=\"0 0 252.646875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:28:54.745221</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 252.646875 180.65625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 143.1 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m716e83ed01\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m716e83ed01\" x=\"76.474554\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 143.1 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m716e83ed01\" x=\"116.331696\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 143.1 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m716e83ed01\" x=\"156.188839\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 143.1 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m716e83ed01\" x=\"196.045982\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m716e83ed01\" x=\"235.903125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 108.403505 \n",
       "L 235.903125 108.403505 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"md12bcb22a8\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#md12bcb22a8\" x=\"40.603125\" y=\"108.403505\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.202724)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 71.520929 \n",
       "L 235.903125 71.520929 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md12bcb22a8\" x=\"40.603125\" y=\"71.520929\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 75.320148)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 34.638352 \n",
       "L 235.903125 34.638352 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md12bcb22a8\" x=\"40.603125\" y=\"34.638352\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 38.437571)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.377273 \n",
       "L 44.588839 16.712104 \n",
       "L 48.574554 22.027161 \n",
       "L 52.560268 29.690011 \n",
       "L 56.545982 38.505764 \n",
       "L 60.531696 49.917184 \n",
       "L 64.517411 57.614152 \n",
       "L 68.503125 62.845605 \n",
       "L 72.488839 65.28806 \n",
       "L 76.474554 67.959173 \n",
       "L 80.460268 69.701745 \n",
       "L 84.445982 71.592525 \n",
       "L 88.431696 74.666191 \n",
       "L 92.417411 76.492509 \n",
       "L 96.403125 79.545545 \n",
       "L 100.388839 80.996475 \n",
       "L 104.374554 83.100221 \n",
       "L 108.360268 86.895745 \n",
       "L 112.345982 88.050467 \n",
       "L 116.331696 90.572993 \n",
       "L 120.317411 92.313589 \n",
       "L 124.303125 94.28507 \n",
       "L 128.288839 95.937169 \n",
       "L 132.274554 98.564997 \n",
       "L 136.260268 99.747939 \n",
       "L 140.245982 102.683802 \n",
       "L 144.231696 104.858001 \n",
       "L 148.217411 107.416149 \n",
       "L 152.203125 109.292467 \n",
       "L 156.188839 112.00577 \n",
       "L 160.174554 114.414351 \n",
       "L 164.160268 117.339765 \n",
       "L 168.145982 119.188174 \n",
       "L 172.131696 121.603342 \n",
       "L 176.117411 122.908656 \n",
       "L 180.103125 125.927947 \n",
       "L 184.088839 127.214444 \n",
       "L 188.074554 128.109978 \n",
       "L 192.060268 130.546138 \n",
       "L 196.045982 132.044746 \n",
       "L 200.031696 132.821754 \n",
       "L 204.017411 133.915646 \n",
       "L 208.003125 133.852694 \n",
       "L 211.988839 134.895592 \n",
       "L 215.974554 135.438857 \n",
       "L 219.960268 136.174952 \n",
       "L 223.945982 136.507532 \n",
       "L 227.931696 136.922727 \n",
       "L 231.917411 136.843601 \n",
       "L 235.903125 135.475167 \n",
       "\" clip-path=\"url(#p34ce236f7a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298437 \n",
       "L 185.628125 20.298437 \n",
       "L 195.628125 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p34ce236f7a\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()\n",
    "num_epochs, lr = 500, 1\n",
    "model = d2l.RNNModelScratch(len(vocab), num_hiddens, device, get_lstm_params,\n",
    "                            init_lstm_state, lstm)\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 24,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 简洁实现\n",
    "\n",
    "### PyTorch的LSTM模块优势\n",
    "\n",
    "使用`nn.LSTM`可以快速构建LSTM模型：\n",
    "\n",
    "**优势**：\n",
    "- **代码简洁**：只需一行定义\n",
    "- **性能优化**：使用C++/CUDA实现，速度更快（通常快10倍以上）\n",
    "- **自动处理**：梯度计算、参数管理等\n",
    "- **功能完整**：支持多层、双向、dropout等\n",
    "\n",
    "**参数说明**：\n",
    "- `num_inputs`：输入维度（词汇表大小）\n",
    "- `num_hiddens`：隐藏单元数\n",
    "\n",
    "**注意**：\n",
    "- PyTorch的LSTM默认返回`(output, (h_n, c_n))`\n",
    "- `h_n`和`c_n`是最后一个时间步的状态\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:28:54.782381Z",
     "iopub.status.busy": "2023-08-18T07:28:54.781811Z",
     "iopub.status.idle": "2023-08-18T07:29:19.488133Z",
     "shell.execute_reply": "2023-08-18T07:29:19.487180Z"
    },
    "origin_pos": 26,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.1, 234815.0 tokens/sec on cuda:0\n",
      "time traveller for so it will be convenient to speak of himwas e\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"180.65625pt\" viewBox=\"0 0 252.646875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:29:19.454029</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 252.646875 180.65625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 143.1 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"maf934e7d81\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf934e7d81\" x=\"76.474554\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 143.1 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf934e7d81\" x=\"116.331696\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 143.1 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf934e7d81\" x=\"156.188839\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 143.1 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf934e7d81\" x=\"196.045982\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf934e7d81\" x=\"235.903125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 106.567198 \n",
       "L 235.903125 106.567198 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"mcfaba2570a\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcfaba2570a\" x=\"40.603125\" y=\"106.567198\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 110.366417)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 68.217917 \n",
       "L 235.903125 68.217917 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcfaba2570a\" x=\"40.603125\" y=\"68.217917\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 72.017136)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 29.868637 \n",
       "L 235.903125 29.868637 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcfaba2570a\" x=\"40.603125\" y=\"29.868637\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 33.667855)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.377273 \n",
       "L 44.588839 21.832772 \n",
       "L 48.574554 35.192916 \n",
       "L 52.560268 52.401494 \n",
       "L 56.545982 60.327059 \n",
       "L 60.531696 64.623995 \n",
       "L 64.517411 69.372083 \n",
       "L 68.503125 72.157991 \n",
       "L 72.488839 75.2356 \n",
       "L 76.474554 79.059318 \n",
       "L 80.460268 83.066156 \n",
       "L 84.445982 84.595659 \n",
       "L 88.431696 89.656091 \n",
       "L 92.417411 92.093826 \n",
       "L 96.403125 95.259217 \n",
       "L 100.388839 98.661807 \n",
       "L 104.374554 101.076422 \n",
       "L 108.360268 105.250063 \n",
       "L 112.345982 107.448831 \n",
       "L 116.331696 110.66592 \n",
       "L 120.317411 114.846328 \n",
       "L 124.303125 118.172503 \n",
       "L 128.288839 120.815058 \n",
       "L 132.274554 124.153129 \n",
       "L 136.260268 126.77078 \n",
       "L 140.245982 129.141869 \n",
       "L 144.231696 130.07523 \n",
       "L 148.217411 132.586366 \n",
       "L 152.203125 133.753072 \n",
       "L 156.188839 134.864893 \n",
       "L 160.174554 134.920052 \n",
       "L 164.160268 136.031106 \n",
       "L 168.145982 135.914991 \n",
       "L 172.131696 136.137546 \n",
       "L 176.117411 136.306486 \n",
       "L 180.103125 136.635479 \n",
       "L 184.088839 136.589523 \n",
       "L 188.074554 136.455455 \n",
       "L 192.060268 136.176927 \n",
       "L 196.045982 136.79973 \n",
       "L 200.031696 136.600568 \n",
       "L 204.017411 136.640853 \n",
       "L 208.003125 136.808802 \n",
       "L 211.988839 136.732722 \n",
       "L 215.974554 136.883154 \n",
       "L 219.960268 136.867975 \n",
       "L 223.945982 136.874055 \n",
       "L 227.931696 136.922727 \n",
       "L 231.917411 135.968865 \n",
       "L 235.903125 136.815677 \n",
       "\" clip-path=\"url(#pbb8d7abbab)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298437 \n",
       "L 185.628125 20.298437 \n",
       "L 195.628125 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pbb8d7abbab\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_inputs = vocab_size\n",
    "lstm_layer = nn.LSTM(num_inputs, num_hiddens)\n",
    "model = d2l.RNNModel(lstm_layer, len(vocab))\n",
    "model = model.to(device)\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 29,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "长短期记忆网络是典型的具有重要状态控制的隐变量自回归模型。\n",
    "多年来已经提出了其许多变体，例如，多层、残差连接、不同类型的正则化。\n",
    "然而，由于序列的长距离依赖性，训练长短期记忆网络\n",
    "和其他序列模型（例如门控循环单元）的成本是相当高的。\n",
    "在后面的内容中，我们将讲述更高级的替代模型，如Transformer。\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 长短期记忆网络有三种类型的门：输入门、遗忘门和输出门。\n",
    "* 长短期记忆网络的隐藏层输出包括“隐状态”和“记忆元”。只有隐状态会传递到输出层，而记忆元完全属于内部信息。\n",
    "* 长短期记忆网络可以缓解梯度消失和梯度爆炸。\n",
    "\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 调整和分析超参数对运行时间、困惑度和输出顺序的影响。\n",
    "1. 如何更改模型以生成适当的单词，而不是字符序列？\n",
    "1. 在给定隐藏层维度的情况下，比较门控循环单元、长短期记忆网络和常规循环神经网络的计算成本。要特别注意训练和推断成本。\n",
    "1. 既然候选记忆元通过使用$\\tanh$函数来确保值范围在$(-1,1)$之间，那么为什么隐状态需要再次使用$\\tanh$函数来确保输出值范围在$(-1,1)$之间呢？\n",
    "1. 实现一个能够基于时间序列进行预测而不是基于字符序列进行预测的长短期记忆网络模型。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的变体\n",
    "\n",
    "多年来提出了许多LSTM变体：\n",
    "- **多层LSTM**：堆叠多个LSTM层，提高表达能力\n",
    "- **双向LSTM**：同时处理前向和后向信息\n",
    "- **残差连接**：缓解深层网络的梯度问题\n",
    "- **不同类型的正则化**：Dropout、权重衰减等\n",
    "- **Peephole连接**：让门直接观察记忆元\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM的计算成本\n",
    "\n",
    "**训练成本高**：\n",
    "- 参数数量多（约为RNN的2倍）\n",
    "- 每个时间步需要计算4组权重矩阵乘法\n",
    "- 训练时间长，内存占用大\n",
    "\n",
    "**为什么仍然值得？**\n",
    "- 能够学习长期依赖\n",
    "- 在许多任务上表现优异\n",
    "- 是深度学习序列模型的重要里程碑\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 现代替代方案\n",
    "\n",
    "**Transformer架构**：\n",
    "- 使用**注意力机制**替代循环结构\n",
    "- 可以并行计算，训练效率高\n",
    "- 更好地处理长距离依赖\n",
    "- 已成为当前NLP的主流架构\n",
    "\n",
    "**学习建议**：\n",
    "- 理解LSTM是理解Transformer的基础\n",
    "- LSTM仍在许多场景中广泛应用\n",
    "- 掌握多种模型，根据任务选择合适架构\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSTM vs RNN对比\n",
    "\n",
    "| 特性 | RNN | LSTM |\n",
    "|------|-----|------|\n",
    "| **参数数量** | 少 | 多（约2倍） |\n",
    "| **梯度消失** | 严重 | 缓解 |\n",
    "| **长期依赖** | 差（<10步） | 好（>100步） |\n",
    "| **计算速度** | 较快 | 较慢 |\n",
    "| **表达能力** | 弱 | 强 |\n",
    "| **门控机制** | 无 | 3个门 |\n",
    "\n",
    "**适用场景**：\n",
    "- RNN：简单序列任务，序列长度短\n",
    "- LSTM：复杂序列任务，需要长期依赖\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 思考问题\n",
    "\n",
    "1. **为什么LSTM需要三个门？能否减少门的数量？**\n",
    "   - 提示：GRU只有2个门，但性能略差\n",
    "\n",
    "2. **遗忘门和输入门的关系是什么？**\n",
    "   - 它们如何协同工作？\n",
    "   - 是否存在竞争或互补关系？\n",
    "\n",
    "3. **记忆元和隐状态的区别是什么？**\n",
    "   - 为什么需要两个状态？\n",
    "   - 能否合并为一个？\n",
    "\n",
    "4. **LSTM如何解决梯度消失问题？**\n",
    "   - 其机制与RNN的梯度裁剪有何不同？\n",
    "   - 为什么记忆元的更新公式是关键？\n",
    "\n",
    "5. **什么时候应该使用LSTM而不是RNN或Transformer？**\n",
    "   - 考虑计算资源、序列长度、任务复杂度\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 练习\n",
    "\n",
    "1. 调整和分析超参数（隐藏单元数、学习率）对运行时间、困惑度和输出顺序的影响。\n",
    "\n",
    "2. 如何更改模型以生成适当的单词，而不是字符序列？\n",
    "   - 提示：修改词汇表和tokenization方法\n",
    "\n",
    "3. 在给定隐藏层维度的情况下，比较GRU、LSTM和RNN的计算成本。\n",
    "   - 注意：训练和推断的成本\n",
    "\n",
    "4. 既然候选记忆元已经使用$\\tanh$确保值范围在$(-1,1)$，为什么隐状态还需要再次使用$\\tanh$？\n",
    "   - 提示：考虑记忆元更新公式中按元素乘法的效果\n",
    "\n",
    "5. 实现一个基于时间序列预测的LSTM模型（如股票价格预测）。\n",
    "   - 提示：修改数据加载和预处理部分\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "required_libs": [],
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
